Human trust overlays for automated data confidence fabrics

ABSTRACT

A social overlay is provided that allows an application to seek human input when performing high-risk decisions. The overlay allows an application to obtain data confidence scores when performing operations and to obtain human confidence scores that can be used to seek input from users with high confidence scores. This allows applications to improve performance and avoid situations where an automated application may make a wrong decision that does not account for the associated risk.

FIELD OF THE INVENTION

Embodiments of the present invention generally relate to data confidencefabrics (DCFs). More particularly, at least some embodiments of theinvention relate to systems, hardware, software, computer-readablemedia, and methods for increasing the confidence in automated algorithmsby helping the decisions with additional input and by preventingsubstantial damage.

BACKGROUND

There are many examples of algorithms or applications that have madevery poor choices. Automated applications, when faced with decisionsthat have ethical, moral, and business implications may make the wrongdecision and the consequences can be very damaging. Consider therepercussions when chatbots interact with consumers using racist orgenocidal language, of deny service based on discriminatory rulesets, orproliferate inaccurate propaganda. Even assuming that the applicationmay be provided with highly trusted data, there is no mechanism thatallows the application to know that the decisions made from this datahave severe moral or ethical consequences.

More specifically, an application that fully trusts the underlying datamay still cause significant damage (e.g., financial, reputational) to acorporation. The application may, for example, violate the privacy ofconsumers or employees, damage corporate reputational integrity, orviolate legal or compliance requirements. Blaming the application for apoor decision is inadequate to remedy the damage.

There may be situations where the output of an application results inthe loss of life. An application may not perform an identificationprocess correctly or may be responsible for lethal autonomous weapons.Making a wrong decision while controlling an autonomous car may resultin loss of life. Actuation commands in a manufacturing setting may alsolead to loss of life.

There is a trend in application development to try and recognize thesegray areas (e.g., ethical, business, moral implications). Even if anapplication is able to recognize a gray area, the solution is typicallyto wait for human authorization. However, this often introduces a delaythat has an associated cost.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which at least some of the advantagesand features of the invention may be obtained, a more particulardescription of embodiments of the invention will be rendered byreference to specific embodiments thereof which are illustrated in theappended drawings. Understanding that these drawings depict only typicalembodiments of the invention and are not therefore to be considered tobe limiting of its scope, embodiments of the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 illustrates an example of a machine data connection fabric;

FIG. 2 illustrates an example of a human data connection fabric;

FIG. 3 illustrates additional detail about the human data connectionfabric;

FIG. 4 illustrates an example of an overlay that allows a machine dataconnection fabric to be connected to a human data connection fabric;

FIG. 5A illustrates an example of an overlay that includes a networkconnection;

FIG. 5B illustrates an example of an overlay that includes sharedhardware between at least two data connection fabrics;

FIG. 5C illustrates an example of a cloud-based overlay that allows anapplication to access multiple data connection fabrics;

FIG. 6 illustrates an example of an application or workload deployed toan overlay connecting data connection fabrics;

FIG. 7 illustrates a flow diagram where an application may seek humaninput when making decisions; and

FIG. 8 further illustrates an example of a flow diagram of anapplication deployed to an overlay connecting data connection fabrics.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

Embodiments of the present invention generally relate to ecosystems suchas data confidence fabrics (DCFs) and to performing trust operationstherein. Embodiments of the invention further relate to DCFs thatfacilitate user input in automated decision applications or algorithms.Embodiments of the invention further relate DCFs that introduce a useroverlay that allows for trusted user input.

DCFs are generally computing systems that gather data from edge devices.The DCFs measure the reliability or trustworthiness of the data beforeforwarding the data (and/or associated measurements/metadata) upstreamfor application consumption. The DCFs are configured to give the data aconfidence score that reflects the trustworthiness of or confidence inthe data. The confidence score allows applications (and users) to have adegree of confidence in their operations.

However, there are certain environments where applications may makedecisions based on the data in the DCF that may have significantconsequences. For example, some decisions may have business criticalconsequences, moral or ethical ramifications, and may result in loss oflife. Embodiments of the invention introduce a user overlay into oravailable to the DCF where the automated algorithms may not want to relyon machine-generated confidence scores.

In one example, a DCF, by way of example and not limitation, may relateto both hardware and/or software and/or services. A DCF is an example ofan architecture and set of services that allow data to be ingested intoa system for use by applications. The DCF adds or provides trustinformation or scores to the data as the data flows through the DCF.Ultimately, the ingested data is associated with a trust or confidencescore that provides a view into the trustworthiness of the data to anapplication or other use.

More particularly, at least some embodiments of the invention relate tosystems, hardware, software, computer-readable media, and methods forimplementing a user overlay or a user confidence overlay in a DCF.

The following disclosure discusses trust insertion in a DCF prior todiscussing the user overlay. As data flows from data sources to storageor to applications in a DCF ecosystem, scores can be attached to orassociated with the data. As the data flows in the DCF, different formsof trust insertion technology handle or process the data. As the data ishandled by various forms of trust insertion technology, the overallscore or ranking (e.g., a confidence or trustworthiness score) of thedata may change. The data scored or ranked in the DCF system may bestored in various locations, such as a data lake, in a datacenter, in adistributed ledger, or the like. The data scored or ranked in the DCFsystem can be made available to one or more applications or otherclients or users.

By ranking or scoring data, an application is able to explore or exploitthe data for potential analysis or consumption. The score or rank of thedata allows an application to understand or account for thetrustworthiness of the data. For example, the confidence score of thedata may have a significant impact on whether the data is actually usedby the application. An application may require a minimum confidencescore or have other requirements related to the confidence score.

For example, an application operating in a secure facility may need touse data that is very trustworthy (have a high confidence score) whiledata that is used by an application to control lights in a home may notneed to be as trustworthy (a lower confidence score is acceptable). ADCF is able to give or associate data with scores from individual trustinsertion technologies that can be combined in multiple ways todetermine a final score or rank that relates to the trustworthiness ofthe data.

FIG. 1 illustrates an example of a computing network that includes orimplements a DCF. FIG. 1 illustrates an example of a computing networkthat includes a DCF 100. The network 10 may include different networktypes including Internet capable networks, edge computing networks,cellular or telecommunications networks or the like or combinationthereof.

FIG. 1 illustrates devices 102, 104 and 106. These devices 102, 104 and106 may generate data that is ingested into the DCF 100. For example,the devices 102, 104 and 106 may be sensors, smartphones, edge devices,or other data sources. The data generated by the devices 102, 104 and106 may depend on various factors and purposes. For example, the device102 may be a smartphone device that is capable of generating a widevariety of data. In one example, location related data generated by thedevice 102 may be useful to the application 150 if the application 150is a map generating application. If the device 104 is a weather sensingdevice, the data produced by the device 104 may be useful to a weatherrelated application such as the application 152. Embodiments of theinvention thus relate to a wide variety of devices, applications, andnetworks.

The data generated at the devices 102, 104 and 106 may be ingestedthrough a gateway, such as the gateways (GWs) 110, 112 and 114. Agateway, by way of example only, may be configured to receive data fromdevices such as edge devices and perform processing on the data. Thegateways 110, 112, and 114 may deliver the data to servers/storage 120and 122. The servers can store the data, perform processing, or thelike. The DCF 100 may be associated with a distribute ledger 130 (e.g.,a multi-cloud distributed ledger) that allows data to be recorded moresecurely. The cloud 140, which may include multiple clouds fromdifferent providers, may also provide other processing, storage, andcomputing resources. The applications 150 and 152 can be distributed inthe network 10 and be part of the DCF 100 and may operate at differentlevels and/or locations.

In this example, the applications 150 and 152 may interact with edgedata from the devices 102, 104 and 106 with knowledge of thetrustworthiness (e.g., a confidence score) of the data being ingestedinto the DCF 100. In each layer of the DCF 100, data and applicationsare being brought together with known confidence scores in this example.

When implementing the DCF 100, trust insertion technologies are examplesof tools that can be used to increase the confidence in edge data.Examples of trust insertion technologies include, but are not limitedto, hardware root of trust capabilities, co-processors and accelerators,digital signatures, identity management, secure ingest software,encryption, immutable storage, data protection policies, distributedledgers, or the like or combination thereof.

The number and type of trust insertion technologies is large and eachmay be associated with a different trust increase. Differences ordisparities in the trustworthiness of these technologies can impact theoverall trustworthiness of the DCF. Some components or technologies arebetter able to insert or add trust to data flowing in the network.

When a DCF is implemented, however, the trustworthiness of the data isimproved. Because the DCF 100 may include multiple trust insertiontechnologies, confidence scores can be changed at multiple locations asthe data is ingested into the DCF 100.

In FIG. 1, data A is generated by the device 102. IoT (Internet ofThings) capable devices sensors, smartphones, computers, tablets, andthe like are examples of the device 102. The data A is ingested into theDCF 100 and flows through the DCF 100. After flowing through the DCF,the data A may be stored in a repository that can be accessed by theapplication 150 (or another application). The data A may also be storedat various locations while flowing through the DCF 100.

The DCF 100 may be implemented on a wide variety of devices andnetworks. When data is collected on an edge of a network, the data mayflow through various levels of hardware environments that have variouslevels of processing, memory, and storage capabilities. From a hardwareperspective, the data may flow from the data-generating device 102 to aserver computer or to a gateway device 110. The server computer orgateway 110 may send the data to another server that is configured toaggregate data from multiple gateways. That server may send the data toa storage environment where the data can be accessed and used byapplications.

In this path, different devices, services, or applications may handle orprocess the data. Typically, each interaction with the data may beassociated with a trust insertion, where trust metadata is inserted withor associated with the ingested data. When the data reaches theapplication or is stored, the data is thus stored or associated with atrust or confidence score and/or metadata used to determine or calculatethe trust or confidence score. The data and/or the associated score maybe scored immutably.

In this example of FIG. 1, each of the devices and/or services and/orapplications that handle the data may adjust or change thetrustworthiness of the data. This is often achieved by contributing tothe score of the data. More specifically, in one example, this isachieved by providing a score that impacts the trustworthiness score orrank of the data. Typically, the scores are cumulative: each trustinsertion technology contributes to the overall confidence score.Embodiments of the invention allow the score or rank to be computedusing more than simple addition. Other formulations may also beimplemented, such as multiplication, addition, weighting, and/orcombination thereof or the like. More specifically, the overallconfidence score can be determined from the individual scores usingaddition, multiplication, weighting, other scoring algorithms, or thelike or combination thereof.

FIG. 1 illustrates examples of trust insertion technologies 160, 162,164, 166 and 168. The trust insertion technologies include, by way ofexample and not limitation: trust technology 160 where the devices aresigning their data (hardware root of trust signatures on device data);trust technology 162 where the ingest software rejectsnon-authorized/non-authenticated application access to the data (strongN-S-E-W authentication/authorization); trust insertion technology 164,where provenance metadata is appended to the data from the devices(provenance metadata attachment such as information about a data captureenvironment); trust insertion technology 168, where data is stored ontoa cryptographically-protected object store (secure immutable scale-outedge persistence); and trust technology 168, where a record of thedata/metadata is registered in a distributed ledger (e.g., registrationof trusted assets in a multi-cloud distributed ledger).

The confidence added by these trust insertion technologies can be storedwith the data or in another location such as a distributed ledger. Asthe data A traverses the DCF 100, scores are added to the data andresults in a confidence score that can be used by an application. Forexample, an application may only use data that has a thresholdconfidence score. The applications can account for the confidence scoresas they execute.

FIG. 2 illustrates another example of a human DCF. An example of a humanDCF is disclosed in U.S. Ser. No. 16/663,965 filed Oct. 25, 2019, whichis incorporated by reference in its entirety. FIG. 2 illustrates anexample of a human DCF 200 that is configured to generate confidencescores associated with users or humans, while FIG. 1 illustrates anexample of a machine DCF 100 that generates confidence scores for data.The DCF 200 is configured to generate confidence scores that can beapplied to or associated with a particular person or user. Further,confidence scores can be generated for multiple characteristics such as,but not limited to: morality skill, knowledge, trustworthiness anddependability.

In the DCF 200, a user 202 may be associated with multiple gateways suchas gateways 204, 206, 208, 210 and 212. These gateways generate outputthat may include data/metadata and a confidence scores 232, representedas outputs 222, 224, 226, 228, and 230. In this example, the user 202may enable trust measurement, the DCF 200 may gather input or monitorthe user and generate confidence scores. An application 220 may measurethe confidence.

More specifically, the gateways 204, 206, 208, 210, 212 may be,respectively, a camera 204 that captures a video of the user 202, a homeassistant 206 (e.g., Amazon Echo) that captures audio data, a personallaptop 208 that may capture keystrokes, an AR/VR device 210 that watchesthe user's 202 skills and reactions as they perform a task, or a mobiledevice 212 that has location-based services enabled. The confidencescores 232 (which may be signed and immutable) are transmitted withinthe DCF 200 to edge servers/storage 214 and 216 and may also be storedin the cloud 218. The confidence scores 232 of the user 202 can be usedby an application.

FIG. 3 further illustrates aspects of a human DCF. The DCF 300 can beadapted to multiple users. FIG. 3 may illustrate a portion of the DCF300 that is specific to a user 302. However, the DCF 300 can accommodatemultiple users and may also be configured to protect user identity.

In one example, confidence generators 340 may be associated with theuser 302. The confidence generators 340 are typically configured tocollect data and generate confidence scores, which may be associatedwith confidence measurements, and other data/metadata. These scores andother data are typically signed.

For example, a calendar/location 304 may be configured to combinemeasurements or input from a calendar and location-based services (e.g.,GPS). This allows the calendar location 304 to generate a confidencescore at least related to dependability 336. The confidence score (CS)338 may identify whether the user 302 is dependable. The laptop 306 mayinclude a set of analytic algorithms that can evaluate the honesty(based on the user's appearance in video), their knowledge 328 based onhow the user 302 speaks, what they type, how articulate they are. Thelaptop 306 may be able to evaluate their overall knowledge, oral andwritten knowledge, or the like. These confidence scores may be added tothe user's knowledge 328 CS 330, the skill 324 CS 326, and theirmorality 320 CS 322.

The cell 308 may also perform analytic functions similar to the laptop306 and may also add measurements such as or related to activity andmotion. The video 310 may be a camera that can observe behavior in alarger setting with more people. Audio 312 may also contributeconfidence scores to the characteristics of the user 302. The vehicle314 may be able to provide information related to whether the user 302is obedient to traffic laws, whether they drive aggressively, or whetherthe user 302 follows suggested commands. The AR/VR/MR (augmentedreality, virtual reality, mixed reality) 316 can follow the user 302 asthe user works through instructions on how to fix or preplace a part andcan evaluate the user's ability to follow directions, timeliness,thoroughness, effectiveness, and the like.

The various confidence generators 340 thus generate confidence scoresthat are accumulated to the confidence scores 322, 326, 330, 334, and338 of, respectively, morality 320, skill 324, knowledge 328,trustworthiness 332, and dependability 336. It is understood that thesecharacteristics are presented by way of example only.

The skill 324 CS 326 may accumulate data from multiple confidencegenerators. As a result, the CS 326 may be an average, a weightedaverage, or the like of confidence scores from multiple generators. Theconfidence scores of the other characteristics 342 may be similarlydetermined. The user 302 may also be associated with an overallconfidence score, which may be a combination of the individualconfidence scores of the characteristics 342.

The DCF 300, which is an example of the DCF 200, is generatingconfidence in the ability of the user 302 to perform certain tasks, makecertain decisions, or solve certain problems including businessproblems. For example, if the DCF 300 is implemented in a factory andassociated with the workers in the factory, the DCF may generateconfidence scores for the workers or users with regard to performingcertain tasks. When a task arises that requires certain input, the DCFor the confidence scores generated by the DCF 300 can be used to selecta specific user and seek input from that specific user. The userselected by an application may vary according to the requirement. Forexample, a highly skilled person may be needed for one task while adependable and trustworthy person may be needed for a different task.

FIG. 4 illustrates an example of a DCF overlay. FIG. 4 illustrates amachine DCF 402 and a human DCF 404. The DCF 402 and the DCF 404 may beassociated with an organization or entity, a geography, or the like. Forexample, the DCF 402 and the DCF 404 may be located in a similargeography such as a manufacturing floor.

The machine DCF 402 may receive or ingest telemetry data from sensors ordevices (e.g., temperature sensors, humidity devices, robot-arms, formanufacturing, or the like). The ingested data is scored as previouslydiscussed. An application that relies on the data ingested and scored bythe DCF 402 can have a level of confidence in the ingested data.However, the ability of the machine DCF 402 to make decisions in grayareas (e.g., ethical, moral, loss of life, business) is limited.

The human DCF 404 generates confidence scores for users or workers inthe manufacturing floor (or other locations). The actions of a human, asillustrated in FIG. 3, are captured and the overall confidence score ortrustworthiness of the workers can be determined by the human DCF 404.

FIG. 4 illustrates a DCF overlay 406 that allows the DCF 402 and the DCF404 to be combined or used together in a beneficial manner. The overlay406, for example, may allow an application to make decisions in the greyarea more quickly and with more confidence. The overlay 406 can increaseconfidence in cases or situations where automated applications may notwant to rely on machine generated confidence scores.

An overlay can be achieved in different ways. FIG. 5A illustrates anexample of an overlap between a machine DCF 560 and a human DCF 562. TheDCF 560 is configured to ingest data from a device 502 and generateconfidence scores 504 and 506 that are associated with the data ingestedfrom the device 502. As previously stated, the data may pass throughvarious layers.

The DCF 562 ingests data or generates data related to a user 520. Theconfidence scores 508 and 510 relate to the user 520 and may be anoverall confidence score or confidence scores for variouscharacteristics of the user 520.

In this example, the application 512 may use the data generated by thedevice in conjunction with the confidence scores 504 and 506. Similarly,the application 514 may use the confidence scores 508 and 510 associatedwith the user 520 in various ways.

FIG. 5A further illustrates an example of creating a social overlaybetween the DCF 560 and the DCF 562. FIG. 5A illustrates and the DCF 560is networked 564 to the DCF 562. In this example, the network connectionenables connectivity between the machine DCF 560 and the human DCF 562.

FIG. 5B illustrates another example of creating a social overlay betweenthe DCF 560 and the DCF 562. In this example, the DCF 560 and the DCF562 share hardware including a gateway 524 and/or a server 522 and orother hardware.

FIG. 5C illustrates another example of creating a social overlay betweenthe DCF 560 and the DCF 562. FIG. 5C illustrates an application orcloud-integrated overlay. This allows an application, such as theapplication 516, to analyze data from more than one DCF. In addition,the application 516 may also send actuation commands.

Workloads that leverage social overlays, such as illustrated in FIGS.5A-5C can be written and deployed into the social overlay. FIG. 6illustrates an example of an application that is deployed into thesocial overlay. FIG. 6 illustrates an overlay 606 between a machine DCF602 and a human DCF 604. An application or workload 610 has beendeployed to the social overlay 606.

The application 610, by being deployed in a social overlay 606 or byhaving access to the social overlay 606, is able to run workloads inwhich trusted information about relevant human actors or users can beused to make fast decisions in a high-risk situation. The application610 can use the trusted information to make decisions in grey areas suchas ethical area, mission critical areas, loss of life areas, and thelike or combination thereof.

FIG. 7 illustrates an example of a process for performing decisionsusing a social overlay that relates one or more DCFs. FIG. 7 isdiscussed with reference to an application that is running within asocial overlay. Because the application is running in the socialoverlay, the application has access to data confidence scores, data, andhuman confidence scores, for example.

FIG. 7 illustrates that an application may access 700 data that has beeningested into or by a machine DCF. In this example, the application 710may access a ledger entry 704 associated with ingested data, such as thedata set 702. The application 710 may also access the confidence score706 (and/or other confidence metadata) and location or object identifier708 from the ledger entry 704.

The application 710 may then access the data set 702 or process the dataset after understanding the confidence score and/or other confidencemetadata (e.g., source, provenance, individual contributions, date,time, etc.) and obtaining the location. The process of accessing datamay occur in the context of a machine DCF.

During execution of the application 710, a decision point may bereached. The application may then determine 712 whether the decision isa high-risk decision or is in a grey area. A determination of No at 712results in the application sending 724 an actuation command orperforming another action in an automated manner.

A determination of Yes at 712 may cause the application to determine 714a decision type. In this example, the decision type may be a missioncritical (e.g., business related, ethical, or loss of life decision).Each type, however, leads to a step of seeking 716, 718, or 720 humaninput.

However, they type of human input that is requested can vary. Morespecifically, the analysis of the decision type may determine that theapplication is looking fora highly skilled person, or a dependableperson, or an honest person, or other characteristic.

Because the social overlay allows access to the human DCF, confidencescores for various users can be obtained. As a result, the applicationmay choose the user that satisfies the requirements of the decisiontype. When accessing the human DCF, the access may also identify thelocation of the user (if available) and the most recently used device orthe device that most recently contributed data to the confidence scoreof the user.

The application, once the user is identified, may solicit feedback invarious manners including, by way of example only, calling the user,texting the user, asking via a nearby audio device, prompting on anAR/VR device, or the like. Social media activity of video can also beanalyzed and may result in a faster decision.

When the human input is received 722, the application can send 724 anactuation when the response is Yes (or permission is granted) or theapplication does not send 726 an actuation when the response is No (orpermission is not granted). This process can be logged such that thedecisions can be traced back to both the machine and human inputs thatwere used to make the decision.

FIG. 8 further illustrates this method from the perspective of a socialoverlay. FIG. 8 illustrates a method where device data is accessed 810into an application. This corresponds to the application 806 accessingthe machine DCF 802 to acquire the data that has been scored by the DCF802 at 1. If a high-risk decision is required in 812, a yes causes theapplication 806 to seek human input 814. This is illustrated by theapplication 806 accessing the human DCF 804 at 2.

In one example, seeking 814 human input may include a search of the userconfidence scores to find a suitable overall confidence score or to finda suitable confidence score for a characteristic. The search for humaninput may also account for factors of whether the user is available(alternative users may also be selected in case a user cannot bereached), methods for contacting the user, location of the user, or thelike or combination thereof.

Seeking 814 human input may also include presenting a query to the useron a display, visually, audibly, text message, email, telephone call, orthe like. Seeking 814 human input may thus include, after identifying auser based on the confidence score, contacting the user.

Next, human input is received 816 at 3. The response may come in thesame communication used to contact the user, via a different method, orthe like. For example, the user may first be authenticated or the likeprior to accepting the received input. If the input is affirmative orapproved (Y as 818), an actuation is sent 822 at 4. If the receivedinput is negative or if permission is not granted (N at 818) theactuation is not sent 820.

Embodiments of the invention, such as the examples disclosed herein, maybe beneficial in a variety of respects. For example, and as will beapparent from the present disclosure, one or more embodiments of theinvention may provide one or more advantageous and unexpected effects,in any combination, some examples of which are set forth below. Itshould be noted that such effects are neither intended, nor should beconstrued, to limit the scope of the claimed invention in any way. Itshould further be noted that nothing herein should be construed asconstituting an essential or indispensable element of any invention orembodiment. Rather, various aspects of the disclosed embodiments may becombined in a variety of ways so as to define yet further embodiments.Such further embodiments are considered as being within the scope ofthis disclosure. As well, none of the embodiments embraced within thescope of this disclosure should be construed as resolving, or beinglimited to the resolution of, any particular problem(s). Nor should anysuch embodiments be construed to implement, or be limited toimplementation of, any particular technical effect(s) or solution(s).Finally, it is not required that any embodiment implement any of theadvantageous and unexpected effects disclosed herein.

The following is a discussion of aspects of example operatingenvironments for various embodiments of the invention. This discussionis not intended to limit the scope of the invention, or theapplicability of the embodiments, in any way.

In general, embodiments of the invention may be implemented inconnection with systems, software, and components, that individuallyand/or collectively implement, and/or cause the implementation of, DCFoperations. Such operations may include, but are not limited to,automated applications using a social overlay to seek human input, dataread/write/delete operations, data deduplication operations, data backupoperations, data restore operations, data cloning operations, dataarchiving operations, and disaster recovery operations. More generally,the scope of the invention embraces any operating environment in whichthe disclosed concepts may be useful.

New and/or modified data collected and/or generated in connection withsome embodiments, may be stored in a data protection environment thatmay take the form of a public or private cloud storage environment, anon-premises storage environment, and hybrid storage environments thatinclude public and private elements. Any of these example storageenvironments, may be partly, or completely, virtualized. The storageenvironment may comprise, or consist of, a datacenter which is operableto service read, write, delete, backup, restore, and/or cloning,operations initiated by one or more clients or other elements of theoperating environment. Where a backup comprises groups of data withdifferent respective characteristics, that data may be allocated, andstored, to different respective targets in the storage environment,where the targets each correspond to a data group having one or moreparticular characteristics.

Example public cloud storage environments in connection with whichembodiments of the invention may be employed include, but are notlimited to, Microsoft Azure, Amazon AWS, and Google Cloud. Moregenerally however, the scope of the invention is not limited toemployment of any particular type or implementation of cloud storage.

In addition to the storage environment, the operating environment mayalso include one or more clients that are capable of collecting,modifying, and creating, data. As such, a particular client may employ,or otherwise be associated with, one or more instances of each of one ormore applications that perform such operations with respect to data.

Devices in the operating environment may take the form of software,physical machines, or virtual machines (VM), or any combination ofthese, though no particular device implementation or configuration isrequired for any embodiment. Similarly, data protection systemcomponents such as databases, storage servers, storage volumes (LUNs),storage disks, replication services, backup servers, restore servers,backup clients, and restore clients, for example, may likewise take theform of software, physical machines or virtual machines (VM), though noparticular component implementation is required for any embodiment.Where VMs are employed, a hypervisor or other virtual machine monitor(VMM) may be employed to create and control the VMs. The term VMembraces, but is not limited to, any virtualization, emulation, or otherrepresentation, of one or more computing system elements, such ascomputing system hardware. A VM may be based on one or more computerarchitectures, and provides the functionality of a physical computer. AVM implementation may comprise, or at least involve the use of, hardwareand/or software. An image of a VM may take various forms, such as a.VMDK file for example.

As used herein, the term ‘data’ is intended to be broad in scope. Thus,that term embraces, by way of example and not limitation, data segmentssuch as may be produced by data stream segmentation processes, datachunks, data blocks, atomic data, emails, objects of any type, files ofany type including media files, word processing files, spreadsheetfiles, and database files, as well as contacts, directories,sub-directories, volumes, and any group of one or more of the foregoing.

Example embodiments of the invention are applicable to any systemcapable of storing and handling various types of objects, in analog,digital, or other form. Although terms such as document, file, segment,block, or object may be used by way of example, the principles of thedisclosure are not limited to any particular form of representing andstoring data or other information. Rather, such principles are equallyapplicable to any object capable of representing information.

As used herein, the term ‘backup’ is intended to be broad in scope. Assuch, example backups in connection with which embodiments of theinvention may be employed include, but are not limited to, full backups,partial backups, clones, snapshots, and incremental or differentialbackups.

Following are some further example embodiments of the invention. Theseare presented only by way of example and are not intended to limit thescope of the invention in any way.

Embodiment 1. A method, comprising: accessing data from a machine dataconfidence fabric, seeking user input based on confidence scores ofusers from a human data confidence fabric with regard to a decision,receiving input from the user regarding the decision, and performing anaction associated with the decision when the input from the user isaffirmative.

Embodiment 2. The method of embodiment 1, further comprising accessingdata and confidence scores associated with the data from the machinedata confidence fabric.

Embodiment 3. The method of embodiment 1 and/or 2, further comprisingdetermining a type of the decision.

Embodiment 4. The method of any of embodiments 1-3, further comprisingdetermining whether to seek overall confidence scores of users orconfidences scores of users that are related to specific characteristicsassociated with the type of the decision.

Embodiment 5. The method of any of embodiments 1-4, further comprisingsending a request to a user whose confidence score is suitable for thedecision.

0080 Embodiment 6. The method of any of embodiments 1-5, furthercomprising, sending the request via at least one of text, telephone,video, or audio.

Embodiment 7. The method of any of embodiments 1-6, further comprisingreceiving a response from the user.

Embodiment 8. The method of any of embodiments 1-7, further comprisingseeking human input based on a location of the user.

Embodiment 9. The method of any of embodiments 1-8, further comprising,creating an overlay to connect the machine data confidence fabric withthe human data confidence fabric.

Embodiment 10. The method of any of embodiments 1-9, wherein the overlaycomprises at least one of: a network connection to connect the machinedata connection fabric with the human data connection fabric, hardwarethat is shared by the machine data connection fabric and the human dataconnection fabric, or a cloud-integrated connection that allows anapplication to analyze data from multiple data connection fabricsincluding the machine data connection fabric and the human dataconnection fabric.

Embodiment 11. The method of any of embodiments 1-10, further comprisingdeploying an application to the overlay.

Embodiment 12. The method as recited in any of embodiments 1-11 or anyportions thereof.

Embodiment 13. A non-transitory storage medium having stored thereininstructions that are executable by one or more hardware processors toperform the operations of any one or more of embodiments 1-12.

The embodiments disclosed herein may include the use of a specialpurpose or general-purpose computer including various computer hardwareor software modules, as discussed in greater detail below. A computermay include a processor and computer storage media carrying instructionsthat, when executed by the processor and/or caused to be executed by theprocessor, perform any one or more of the methods disclosed herein, orany part(s) of any method disclosed.

As indicated above, embodiments within the scope of the presentinvention also include computer storage media, which are physical mediafor carrying or having computer-executable instructions or datastructures stored thereon. Such computer storage media may be anyavailable physical media that may be accessed by a general purpose orspecial purpose computer.

By way of example, and not limitation, such computer storage media maycomprise hardware storage such as solid state disk/device (SSD), RAM,ROM, EEPROM, CD-ROM, flash memory, phase-change memory (“PCM”), or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other hardware storage devices which may be used tostore program code in the form of computer-executable instructions ordata structures, which may be accessed and executed by a general-purposeor special-purpose computer system to implement the disclosedfunctionality of the invention. Combinations of the above should also beincluded within the scope of computer storage media. Such media are alsoexamples of non-transitory storage media, and non-transitory storagemedia also embraces cloud-based storage systems and structures, althoughthe scope of the invention is not limited to these examples ofnon-transitory storage media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed, cause a general purpose computer, specialpurpose computer, or special purpose processing device to perform acertain function or group of functions. As such, some embodiments of theinvention may be downloadable to one or more systems or devices, forexample, from a website, mesh topology, or other source. As well, thescope of the invention embraces any hardware system or device thatcomprises an instance of an application that comprises the disclosedexecutable instructions.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts disclosed herein are disclosed asexample forms of implementing the claims.

As used herein, the term ‘module’ or ‘component’ may refer to softwareobjects or routines that execute on the computing system. The differentcomponents, modules, engines, and services described herein may beimplemented as objects or processes that execute on the computingsystem, for example, as separate threads. While the system and methodsdescribed herein may be implemented in software, implementations inhardware or a combination of software and hardware are also possible andcontemplated. In the present disclosure, a ‘computing entity’ may be anycomputing system as previously defined herein, or any module orcombination of modules running on a computing system.

In at least some instances, a hardware processor is provided that isoperable to carry out executable instructions for performing a method orprocess, such as the methods and processes disclosed herein. Thehardware processor may or may not comprise an element of other hardware,such as the computing devices and systems disclosed herein.

In terms of computing environments, embodiments of the invention may beperformed in client-server environments, whether network or localenvironments, or in any other suitable environment. Suitable operatingenvironments for at least some embodiments of the invention includecloud computing environments where one or more of a client, server, orother machine may reside and operate in a cloud environment.

With reference briefly now to the Figures, any one or more of theentities disclosed, or implied, by the Figures and/or elsewhere herein,may take the form of, or include, or be implemented on, or hosted by, aphysical computing device, one example of which is denoted at. As well,where any of the aforementioned elements comprise or consist of avirtual machine (VM), that VM may constitute a virtualization of anycombination of the physical components disclosed herein.

In one example, the physical computing device includes a memory whichmay include one, some, or all, of random access memory (RAM),non-volatile random access memory (NVRAM), read-only memory (ROM), andpersistent memory, one or more hardware processors, non-transitorystorage media, UI device, and data storage. One or more of the memorycomponents of the physical computing device may take the form ofsolid-state device (SSD) storage. As well, one or more applications maybe provided that comprise instructions executable by one or morehardware processors to perform any of the operations, or portionsthereof, disclosed herein.

Such executable instructions may take various forms including, forexample, instructions executable to perform any method or portionthereof disclosed herein, and/or executable by/at any of a storage site,whether on-premises at an enterprise, or a cloud storage site, client,datacenter, or backup server, to perform any of the functions disclosedherein. As well, such instructions may be executable to perform any ofthe other operations and methods, and any portions thereof, disclosedherein including, but not limited to DCF operations and operations thatseek human input.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A method, comprising: operating an application,wherein the application accesses data from a machine data confidencefabric, wherein the machine data confidence fabric associates confidencescores with data ingested into the machine data confidence fabric;reaching a decision point in the application and determining that thedecision point is mission critical that requires input from a user,wherein the decision point is associated with a decision type;accessing, by the application, a human data confidence fabric andselecting, by the application, a user that satisfies requirements of thedecision type, wherein the requirements include a confidence score abovea threshold confidence score, wherein the human data confidence fabricgenerates confidence scores for users; seeking user input from theselected user with regard to a decision to be made at the decisionpoint; receiving input from the selected user regarding the decision;and performing an action associated with the decision when the inputfrom the user is affirmative.
 2. The method of claim 1, furthercomprising accessing the data and the confidence scores associated withthe data from the machine data confidence fabric.
 3. The method of claim1, further comprising determining the decision type.
 4. The method ofclaim 3, further comprising determining whether to seek overallconfidence scores of users or confidences scores of users that arerelated to specific characteristics associated with the decision type.5. The method of claim 1, further comprising sending a request toselected user.
 6. The method of claim 5, further comprising, sending therequest via at least one of text, telephone, video, or audio.
 7. Themethod of claim 6, further comprising receiving a response from theselected user.
 8. The method of claim 1, further comprising selectingthe user based on a location of the user.
 9. The method of claim 1,further comprising, creating an overlay to connect the machine dataconfidence fabric with the human data confidence fabric.
 10. The methodof claim 9, wherein the overlay comprises at least one of: a networkconnection to connect the machine data connection fabric with the humandata connection fabric; hardware that is shared by the machine dataconnection fabric and the human data connection fabric; or acloud-integrated connection that allows an application to analyze datafrom multiple data connection fabrics including the machine dataconnection fabric and the human data connection fabric.
 11. The methodof claim 10, further comprising deploying the application to theoverlay.
 12. A non-transitory storage medium having stored thereininstructions that are executable by one or more hardware processors toperform operations comprising: operating an application, wherein theapplication accesses data from a machine data confidence fabric, whereinthe machine data confidence fabric associates confidence scores withdata ingested into the machine data confidence fabric; reaching adecision point in the application and determining that the decisionpoint is mission critical that requires input from a user, wherein thedecision point is associated with a decision type; accessing, by theapplication, a human data confidence fabric and selecting, by theapplication, a user that satisfies requirements of the decision type,wherein the requirements include a confidence score above a thresholdconfidence score wherein the human data confidence fabric generatesconfidence scores for users; seeking user input from the selected userwith regard to a decision to be made at the decision point; receivinginput from the selected user regarding the decision; and performing anaction associated with the decision when the input from the user isaffirmative.
 13. The non-transitory storage medium of claim 12, furthercomprising accessing the data and confidence scores associated with thedata from the machine data confidence fabric.
 14. The non-transitorystorage medium of claim 12, further comprising determining the decisiontime and determining whether to select the user based on overallconfidence scores of users or confidences scores of users that arerelated to specific characteristics associated with the decision type.15. The non-transitory storage medium of claim 12, further comprisingsending a request to the selected user.
 16. The non-transitory storagemedium of claim 15, further comprising, sending the request via at leastone of text, telephone, video, or audio.
 17. The non-transitory storagemedium of claim 15, further comprising receiving a response from theselected user.
 18. The non-transitory storage medium of claim 12,further comprising selecting the user based on a location of the user.19. The non-transitory storage medium of claim 12, further comprising,creating an overlay to connect the machine data confidence fabric withthe human data confidence fabric and deploying an application to theoverlay.
 20. The non-transitory storage medium of claim 19, wherein theoverlay comprises at least one of: a network connection to connect themachine data connection fabric with the human data connection fabric;hardware that is shared by the machine data connection fabric and thehuman data connection fabric; or a cloud-integrated connection thatallows an application to analyze data from multiple data connectionfabrics including the machine data connection fabric and the human dataconnection fabric.